Techniques for identifying hazardous site conditions in geo-localized enhanced floor plans

ABSTRACT

A system and method for identifying hazardous site conditions. The method includes training a classifier using a labeled training set, wherein the classifier is trained to classify site conditions based on features extracted from enhanced floor plans, wherein the labeled training set includes a plurality of training features and a plurality of training labels, wherein the plurality of training features are extracted from a plurality of training enhanced floor plans, wherein the plurality of training labels are a plurality of hazardous site condition identification labels; and identifying at least one hazardous site condition of a test enhanced floor plan by applying the classifier to a plurality of test features extracted from the test enhanced floor plan, wherein the classifier outputs at least one site condition classification when applied to the plurality of test features, wherein the at least one hazardous site condition is identified based on the at least one site condition classification.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of U.S. Provisional Application No.62/711,069 filed on Jul. 27, 2018. This application is also acontinuation-in-part of U.S. patent application Ser. No. 16/220,709filed on Dec. 14, 2018, now pending, which claims the benefit of U.S.Provisional Application No. 62/598,670 filed on Dec. 14, 2017.

The contents of the above-referenced applications are herebyincorporated by reference.

TECHNICAL FIELD

The present disclosure relates generally to machine learning, and morespecifically to identifying hazardous site conditions such as defectsand safety hazards in enhanced floor plans using machine learningtechniques.

BACKGROUND

During construction of a building, builders such as contractors,construction managers, insurers, construction lenders, and safetyconsultants are interested in monitoring development, ensuring thesafety of workers, and assessing the risks workers are taking at alltimes. To this end, such builders regularly conduct walkthroughinspections (also known as walkthroughs) in which defects, safetyhazards, and other hazardous site conditions that may serve asimpediments to project completion may be identified. A typicalwalkthrough requires substantial attention from builders, and istherefore subject to human error that may result in failure to identifybuilding deficiencies. Any deficiencies during construction musttypically be addressed as soon as possible. As a result, failure toidentify such deficiencies early in construction can result insignificant expenditures to subsequently correct deficiencies, or mayeven result in harm to workers or building occupants.

Due to the need to ensure safety, walkthroughs are conducted on aregular basis such as, for example, daily, weekly, or monthly. Frequentwalkthroughs may require significant devotion of resources due to theattention needed to completely identify all deficiencies. Due to thesepractical limitations, developers, insurers, and owners do not inspectand document site progress as frequently as would be ideal.Additionally, results of the walkthrough may need to be mapped to anappropriate floor plan manually by the person conducting thewalkthrough, which requires significantly more time and may result inmore errors.

To improve efficiency of site inspection, site inspectors may beprovided with a reality capture walkthrough to be evaluated forhazardous site conditions rather than personally conductingwalkthroughs. The site inspector virtually walks the site using mappedimages and identifies hazardous site conditions shown in them. Tofurther aid in making the inspections more efficient, some existingsolutions have been developed to identify defects and hazardous siteconditions in the images using computer imaging. However, both themanual and computer imaging solutions face challenges due to the lowquality or incomplete nature of images that are often provided.Moreover, lack of having a sense of dimension and location adds to theincompleteness of data. As an example, when the images are provided byworkers at the site, the images may provide an incomplete view of thesite due to missing viewpoints and lack of measurement capabilities. Asa result, the existing solutions may fail to accurately identifypotentially hazardous site conditions.

Additionally, the existing computer imaging solutions analyze imageswith respect to a predetermined list of hazardous site conditions, andtherefore cannot adapt to new types of hazardous site conditions withoutexplicit programming by the operator of the computer imaging system.This can be problematic when, for example, new visual identifiers ofpotentially hazardous site conditions appear in the images or new safetyregulations go into effect. Moreover, such solutions typically can onlyidentify objects that could be of interest to site inspectors, but donot account for whether the circumstances surrounding the objectsactually suggest potentially hazardous site conditions. As an example,some existing solutions identify people, hardhats, ladders, and otherobjects that may be of interest to a site inspector, but do notdetermine whether the identified objects demonstrate potentiallyhazardous site conditions. This stems from incomplete images, missingviewpoints, and lack of dimension and location information.

It would therefore be advantageous to provide a solution that wouldovercome the challenges noted above.

SUMMARY

A summary of several example embodiments of the disclosure follows. Thissummary is provided for the convenience of the reader to provide a basicunderstanding of such embodiments and does not wholly define the breadthof the disclosure. This summary is not an extensive overview of allcontemplated embodiments, and is intended to neither identify key orcritical elements of all embodiments nor to delineate the scope of anyor all aspects. Its sole purpose is to present some concepts of one ormore embodiments in a simplified form as a prelude to the more detaileddescription that is presented later. For convenience, the term “someembodiments” or “certain embodiments” may be used herein to refer to asingle embodiment or multiple embodiments of the disclosure.

Certain embodiments disclosed herein include a method for identifyinghazardous site conditions. The method comprises: training a classifierusing a labeled training set, wherein the classifier is trained toclassify site conditions based on features extracted from enhanced floorplans, wherein the labeled training set includes a plurality of trainingfeatures and a plurality of training labels, wherein the plurality oftraining features are extracted from a plurality of training enhancedfloor plans, wherein the plurality of training labels are a plurality ofhazardous site condition identification labels; and identifying at leastone hazardous site condition of a test enhanced floor plan by applyingthe classifier to a plurality of test features extracted from the testenhanced floor plan, wherein the classifier outputs at least one sitecondition classification when applied to the plurality of test features,wherein the at least one hazardous site condition is identified based onthe at least one site condition classification.

Certain embodiments disclosed herein also include a non-transitorycomputer readable medium having stored thereon causing a processingcircuitry to execute a process, the process comprising: training aclassifier using a labeled training set, wherein the classifier istrained to classify site conditions based on features extracted fromenhanced floor plans, wherein the labeled training set includes aplurality of training features and a plurality of training labels,wherein the plurality of training features are extracted from aplurality of training enhanced floor plans, wherein the plurality oftraining labels are a plurality of hazardous site conditionidentification labels; and identifying at least one hazardous sitecondition of a test enhanced floor plan by applying the classifier to aplurality of test features extracted from the test enhanced floor plan,wherein the classifier outputs at least one site conditionclassification when applied to the plurality of test features, whereinthe at least one hazardous site condition is identified based on the atleast one site condition classification.

Certain embodiments disclosed herein also include a system foridentifying hazardous site conditions. The system comprises: aprocessing circuitry; and a memory, the memory containing instructionsthat, when executed by the processing circuitry, configure the systemto: train a classifier using a labeled training set, wherein theclassifier is trained to classify site conditions based on featuresextracted from enhanced floor plans, wherein the labeled training setincludes a plurality of training features and a plurality of traininglabels, wherein the plurality of training features are extracted from aplurality of training enhanced floor plans, wherein the plurality oftraining labels are a plurality of hazardous site conditionidentification labels; and identify at least one hazardous sitecondition of a test enhanced floor plan by applying the classifier to aplurality of test features extracted from the test enhanced floor plan,wherein the classifier outputs at least one site conditionclassification when applied to the plurality of test features, whereinthe at least one hazardous site condition is identified based on the atleast one site condition classification.

Certain embodiments disclosed herein also include a method foridentifying hazardous site conditions. The method comprises: identifyingat least one hazardous site condition of a test enhanced floor plan byapplying a classifier to a plurality of test features extracted from thetest enhanced floor plan, wherein the classifier is trained to classifysite conditions using a labeled training set, wherein a labeled trainingset includes a plurality of training features and a plurality oftraining labels, wherein the plurality of training features areextracted from a plurality of training enhanced floor plans, wherein theplurality of training labels are a plurality of hazardous site conditionidentification labels, wherein the classifier outputs at least one sitecondition classification when applied to the plurality of test features,wherein the at least one hazardous site condition is identified based onthe at least one site condition classification.

BRIEF DESCRIPTION OF THE DRAWINGS

The subject matter disclosed herein is particularly pointed out anddistinctly claimed in the claims at the conclusion of the specification.The foregoing and other objects, features, and advantages of thedisclosed embodiments will be apparent from the following detaileddescription taken in conjunction with the accompanying drawings.

FIG. 1 is a network diagram utilized to describe the various disclosedembodiments.

FIG. 2 is a flow diagram illustrating a training phase for training ahazardous site condition identification model according to anembodiment.

FIG. 3 is a flowchart illustrating a method for identifying hazardoussite conditions according to an embodiment.

FIG. 4 is a schematic diagram of a hazard identifier according to anembodiment.

FIG. 5 is a flowchart illustrating a method for creating an enhancedfloor plan according to an embodiment.

FIGS. 6A-6C are screenshots illustrating views of a graphical userinterface with respect to an enhanced floor plan.

DETAILED DESCRIPTION

It is important to note that the embodiments disclosed herein are onlyexamples of the many advantageous uses of the innovative teachingsherein. In general, statements made in the specification of the presentapplication do not necessarily limit any of the various claimedembodiments. Moreover, some statements may apply to some inventivefeatures but not to others. In general, unless otherwise indicated,singular elements may be in plural and vice versa with no loss ofgenerality. In the drawings, like numerals refer to like parts throughseveral views.

The various disclosed embodiments include a method and system foridentifying hazardous site conditions using machine learning based onfeatures extracted from enhanced floor plans. A labeled training set isfed to a machine learning algorithm to train a classifier to identifyhazardous site conditions. The training set includes features extractedfrom enhanced floor plans and hazard identification labels extractedfrom annotations associated with the enhanced floor plans. The trainingset may include visual features extracted from images of sites,environmental features extracted from other sensor data captured at thesites, or both. The classifier may be trained to classify sets offeatures (each set representing, for example, an object or condition onthe site) into hazardous or non-hazardous, a particular type of hazard,a severity, a likelihood of injury, or a combination thereof.

After training of the classifier, hazardous site conditions may beidentified using the classifier with respect to an enhanced floor plan.To this end, features including visual features, environmental features,or both, are extracted from the enhanced floor plan. Sets of thefeatures are input to the classifier, and the classifier returns aclassification indicating whether the set demonstrates a hazard ordefect, a type of hazard or defect, a severity of each hazard, alikelihood of injury due to each hazard, or a combination thereof. Thesets of features may be grouped with respect to, for example, locations(e.g., points or groups of points) on the enhanced floor plan. Based onthe returned classification, hazardous site conditions are identified. Anotification indicating locations of the identified hazardous siteconditions, a type of each identified hazardous site condition, or both,may be sent to, for example, a user device of a site inspector.

Each enhanced floor plan may be created based on sensor data captured ata site. To this end, the creation of an enhanced floor plan may includegenerating a sparse 3D model including a cloud of 3D points based on thevisual multimedia content and converting the sparse 3D model into a 2Dsite model using machine vision techniques. The 2D site model isgeo-localized, mapped to, and superimposed on a 2D floor plan model. Theenhanced floor plans may be interacted with via a graphical userinterface, thereby allowing for addition of annotations by users. Labelsto be input to the machine learning algorithm may be extracted from theannotations.

FIG. 1 shows an example network diagram 100 utilized to describe thevarious disclosed embodiments. In the example network diagram 100, ahazard identifier 120, a user device 130, a database 140, and datasources 150-1 through 150-N (hereinafter referred to individually as adata source 150 and collectively as data sources 150, merely forsimplicity purposes) are communicatively connected via a network 110.The network 110 may be, but is not limited to, a wireless, cellular orwired network, a local area network (LAN), a wide area network (WAN), ametro area network (MAN), the Internet, the worldwide web (WWW), similarnetworks, and any combination thereof.

The hazard identifier 120 typically includes a processing circuitry anda memory (e.g., the processing circuitry 410 and the memory 415, FIG.4), and is configured to train a classifier and to identify hazardoussite conditions in enhanced floor plans according to the embodimentsdisclosed herein. The hazardous site conditions include defects or otherhazards that may impede progress of site work, present a danger toworkers, both, and the like.

The user device 130 may be, but is not limited to, a personal computer,a laptop, a tablet computer, a smartphone, a wearable computing device,or any other device capable of capturing, storing, and sendingunstructured data sets. The data sources 150 may store data includinginformation related to hazards such as, for example, potential code orother standard violations. As an example, the data sources 150 mayinclude servers of regulatory agencies such as the Occupational Safetyand Health Administration (OSHA).

The database 140 may store enhanced floor plans. The enhanced floorplans may be created by, for example, the floor plan enhancer describedfurther in the above-referenced U.S. patent application Ser. No.16/220,709, assigned to the common assignee, the contents of which arehereby incorporated by reference. The enhanced floor plans are 2D sitemodels superimposed on 2D floor plan models that may include visualmultimedia content elements associated with points on the 2D site model,and environmental parameters associated with points on the 2D sitemodel, or both.

The visual multimedia content may be captured during a walkthrough of asite and mapped to a floor plan during creation of the enhanced floorplan. In some implementations, the capturing of the visual multimediacontent may be performed by a robot configured to perform a walkthroughof the site to ensure that sufficient views of images or video arecaptured. The walkthrough may be, but is not necessarily, along apredetermined walkthrough path. The environmental parameters mayinclude, but are not limited to, dimension data and location data, andsensor signals captured by sensors deployed at the respective sitesrepresented by the enhanced floor plans. The sensor signals mayindicate, for example, noise level, volatile organic compound (VOC)level, temperature, humidity, smoke, and dust particles. The sensors maybe deployed by, for example, being mounted on a robot conducting awalkthrough of the site, being transported by a person conducting awalkthrough, and the like.

The dimension data is a scaled 3d reconstruction sparse model. Thelocation data identifies where objects are in the sparse model (e.g.,with respect to the floor plan). The sparse 3D model is a 3Drepresentation of the site having a point cloud including referencepoints representing portions of any objects or other visual featuresillustrated in the visual multimedia content.

The disclosed embodiments allow for identifying hazardous siteconditions using enhanced floor plans created based on sensor data suchas visual multimedia content and other sensor signals captured at sites.The enhanced floor plans may include the dimensions and distancesbetween the objects within the scene using locations with respect to afloorplan. The enhanced floor plans indicate hazardous conditions at ascene, which may include conditions that are hazardous generally (e.g.,a known slippery substance on the ground), or hazardous when certainconditions are met (e.g., when a known hazardous object is locatedwithin a threshold distance of another object). The known hazardousobject or slippery substance may be, for example, a predetermined objector substance identified in visual multimedia content. As a non-limitingexample, hazardous site conditions identified for fall protectioninclude “Unprotected sides and edges.” Each employee on awalking/working surface with an unprotected side or edge which is 6 feet(1.8 meters) or more above a lower level shall be protected from fallingby the use of guardrail systems, safety net systems, or personal fallarrest systems. If a person is identified as being less than 6 feet fromsuch an unprotected side or edge, the site condition is indicated on theenhanced floor plan as hazardous.

In some implementations, points on an enhanced floor plan may beassociated with 3D visual representations of the respective point. As anon-limiting example, each point on the enhanced floor plan may be aportion of the site occupied during a walkthrough of the site that isassociated with a 3D image captured from that portion of the site. Agraphical user interface including the enhanced floor plan may allowusers to interact with a point on the enhanced floor plan, therebytriggering a display of the 3D visual representation associated with thepoint.

Each enhanced floor plan may further include one or more temporalvariations. Each temporal variation may demonstrate the state of thesite at a different point in time. Accordingly, the enhanced floor planmay allow for viewing historical representations of the site over time(e.g., for the duration of construction). Each point of the enhancedfloor plan may also be associated with historical versions of that pointand their respective annotations. Historical data may be fed to themachine learning algorithm, which may result in a more accurateclassifier due to learning of changes in conditions in a site thatresulted in identification of hazardous site conditions by siteinspectors. As a non-limiting example, when a first point on a siteincludes an image showing a clean floor with no associated annotations,and a subsequent point on the site includes an image showing a floorcovered in a liquid substance with an associated annotation indicating a“housekeeping problem,” the change in annotation may cause learning ofthe liquid substance as a potentially hazardous site condition.

A graphical user interface including the enhanced floor plan may furtherallow users to provide the annotations with respect to points in the 2Dsite model. The annotations may include, but are not limited to,markings on visual multimedia content, text, voice data, links toparticular violations, or a combination thereof. The text or voice datamay indicate, for example, whether a site condition shown in visualmultimedia content is a hazardous site condition, a type of thehazardous site condition, a code or other standard violated by thehazardous site condition, or a combination thereof. As an example, theannotations may indicate that 29 CFR § 1910.305(a)(1)(i) has not beenmet (an OSHA violation) when an image, 3D data, and other data collectedthrough various sensors of the point do not show effective bonding for agrounding connector. The markings may include, for example, boundaryboxes or other boundaries encapsulating specific portions of visualmultimedia content illustrating hazards, 3D segmentations that cover theentire 3D area around the issue, and the like.

In an embodiment, the hazard identifier 120 is configured to utilizemachine learning techniques in order to identify hazardous siteconditions in an enhanced floor plan based on hazardous site conditionsindicated in annotations of training enhanced floor plans. In an exampleimplementation, features extracted from the enhanced floor plans andassociated identifications of issues (i.e., as indicated in theannotations) may be utilized as inputs to a supervised learning processin order to generate a model for predicting whether each portion ofvisual multimedia content element indicates a hazardous site condition,which hazardous site conditions is indicated in each portion of thevisual multimedia content, a severity of each hazardous site condition,a likelihood that injury will occur due to each hazardous sitecondition, a combination thereof, and the like.

To this end, in an embodiment, the hazard identifier 120 is configuredto extract, from training enhanced floor plans, features and labels tobe utilized for training a classifier. In some implementations, onlyfeatures and labels associated with points on the training enhancedfloor plans indicated as hazardous (e.g., as indicated in theannotations) may be extracted.

In an embodiment, the features include visual features, environmentalfeatures, or both. The visual features may include, but are not limitedto, 3D objects, transformations of 2D pixels, transformations of 2Dobjects, a combination thereof, and the like. The environmental featuresmay include, but are not limited to, dimensions, locations, values ofsensor signals, combinations thereof, and the like. The environmentalfeatures may include spatial data, location data, sensor data, and thelike. In a further embodiment, only objects and pixels related tohazardous site conditions (e.g., objects and pixels shown in visualmultimedia content associated with annotations indicating hazardous siteconditions) may be used for the extraction to improve efficiency andaccuracy of training the classifier.

In a further embodiment, extracting the features may further includeextracting 3D objects from a 3D site model of each training enhancedfloor plan. To this end, any 2D content of the visual multimedia contentmay be matched to a corresponding portion of the 3D site model of theenhanced floor plan. Thus, any matched portions of the enhanced floorplan may include the entire (2D and 3D) visual data in addition to text,sounds, volatile organic compounds (VOC) data, dust particle data, andother sensor data, may be categorized as site conditions, and thefeatures of each site condition may be grouped accordingly. Matching the2D visual multimedia content to the 3D site model further allows formore accurate classification by providing information related to threedimensional aspects of site conditions.

In an embodiment, extracting the labels from the annotations includeextracting markings, and may further include extracting key text. Thus,the labels demonstrate, for example, whether portions of the visualmultimedia content element show hazardous site conditions, types ofhazardous site conditions shown, or both. Each extracted labelcorresponds to a subset of the extracted features, for example withrespect to a point on a 3D site model. The extracted markingsdemonstrate portions of the visual multimedia content element showing asite condition determined to be hazardous.

The extracted key text may include, for example, indicators of hazard ornon-hazard status, code numbers, violation names, site conditions,semantics, and the like. In a further embodiment, extracting the labelsmay include pre-processing voice data by, for example, translating thevoice data into text using a natural language processor (not shown). Thekey text may be predetermined words or phrases, words or phrases learnedthrough a machine learning algorithm, and the like.

The features and labels extracted from the training enhanced floor plansare input to a machine learning algorithm to train a classifier toclassify subsets of the features and labels with respect to whether asite condition is hazardous, a type of defect or hazard presented by asite condition, or both (e.g., by classifying into types of hazardoussite conditions including a “no hazard” type). The subsets may includefeatures extracted from portions of the enhanced floor plans associatedfrom the same point (e.g., images and annotations of a specific locationwithin the site).

Alternatively or collectively, the classifier may be trained to classifysite conditions based on likelihood of accident, severity of hazard, orboth. In such an embodiment, the training enhanced floor plans includeannotations indicating likelihood of accidents, severity of hazard, orboth, respectively, thereby allowing for learning site conditions thatare more severe (i.e., more dangerous) or otherwise increase thelikelihood of an accident occurring on site.

In an embodiment, the hazard identifier 120 is configured to inputfeatures extracted from subsequent enhanced floor plans to theclassifier such that the classifier outputs a classification of a sitecondition of each subset of the extracted features. Based on the outputclassifications, the hazard identifier 120 is configured to identify oneor more hazardous site conditions in the enhanced floor plan. In afurther embodiment, the hazard identifier 120 may be configured to send,to the user device 130, a notification indicating the identifiedhazardous site conditions. The notification may further indicate, forexample, the location of each hazardous site condition in the site, acode or other standard associated with the identified hazardous sitecondition, and the like.

It should be noted that the embodiments described herein above withrespect to FIG. 1 are described with respect to one user device 130merely for simplicity purposes and without limitation on the disclosedembodiments. Multiple user devices may be equally utilized withoutdeparting from the scope of the disclosure, thereby allowing forcreating notifying different users of potentially hazardous siteconditions identified in respective enhanced floor plans.

FIG. 2 is a flow diagram 200 illustrating a training phase according toan embodiment. A training set 210 is fed to a machine learning algorithm220 to generate a classifier 230. The training set 210 includessequences of training inputs. The sequences of training inputs includetraining labels 212 as well as training features such as training visualfeatures 211, training environmental features 213, or both.

The training labels 212 include indications of whether correspondingsubsets of the training visual features 211 are hazardous, a type ofhazardous site condition, a severity of each hazardous site condition, alikelihood of accident due to the hazardous site condition, and thelike. To this end, the training labels 212 may include, but are notlimited to, markings on visual multimedia content, key text, or both.

The training visual features 211 include features extracted from theenhanced floor plan such as transformations of 2D pixels and objects, 3Dobjects, dimensions, ambient environmental conditions, and locations.The training environmental features 213 indicate environmentcharacteristics of the site such as, but are not limited to, noiselevel, volatile organic compound (VOC) level, temperature, humidity,smoke, and dust particles. The training environmental features 213 maybe, for example, captured by sensors deployed at the site.

The machine learning algorithm 220 is configured to output theclassifier 230. The classifier 230 classifies subsets of the trainingset 210. To this end, the classifier 230 may be a binary classifier thatclassifies each subset of the training set 210 into either hazardous ornot hazardous. In some implementations, the classifier 230 may be anon-binary classifier that classifies each subset of the training set210 into a specific type of hazardous site condition (e.g., a violationof a particular code or standard practice), a degree of severity (e.g.,high, medium, or low), or a likelihood of accident (e.g., high, medium,low).

FIG. 3 is a flowchart illustrating a method for identifying hazardoussite conditions using machine learning based on an enhanced floor planaccording to an embodiment. In an embodiment, the method is performed bythe hazard identifier 120, FIG. 1.

At S310, training enhanced floor plans are received. The trainingenhanced floor plans may be created, for example, as described hereinbelow with respect to FIG. 5. Each training enhanced floor plan includesvisual multimedia content, environmental data, or both. Each trainingenhanced floor plan also includes annotations associated with therespective visual multimedia content, environmental data, or both.

The visual multimedia content shows different locations on a site (e.g.,locations associated with points on an enhanced floor plan). Theenvironmental data indicates information related to the environment ofthe site such as, but not limited to, dimensions (i.e., data indicatingdistances in the visual multimedia content), locations (e.g., locationsof points with respect to the rest of the floor plan), heat, humidity,smoke.

The annotations in each training enhanced floor plan may includemarkings on visual multimedia content (e.g., a bounding box, cubes, orcircle drawn around a portion of visual multimedia content), text, voicecontent, pointers to other locations (e.g., links to regulatory codes orother external documents indicating potential violations), and the like.

The visual multimedia content element markings may indicate specificportions of visual multimedia content that demonstrate the hazardoussite condition. For example, when the hazardous site condition is amissing guardrail, the markings may include drawing a boundary box wherea guardrail should be. The markings may assist in more accurateidentification of hazardous site conditions by allowing for morespecific visual features to be provided to the classifier.

The text or voice content may be provided via inputs by a user (e.g., asite inspector), and may indicate, for example, whether a point on theenhanced floor plan demonstrates a hazardous or non-hazardous sitecondition, a type of violation (e.g., a code or standard that has beenviolated), the specific hazardous condition (e.g., an object or set ofcircumstances that result in a violation), a combination thereof, andthe like.

At S320, features and corresponding labels are extracted from thetraining enhanced floor plans. The features include visual features,environmental features, or both, as described herein above. The labelsinclude visual indicators (e.g., markings) of hazards, text, voice data,pointers, or a combination thereof. The features and correspondinglabels may be grouped, for example, with respect to respective portionsof the enhanced floor plan (e.g., points on the enhanced floor plan).Each grouping of features and labels is a subset of the features andlabels characterized as a site condition.

In an embodiment, extracting the features, labels, or both, may includepre-processing portions of the enhanced floor plan from which featuresand labels are to be extracted. For extraction of visual features, thepre-processing may include matching 2D visual multimedia content to a 3Dsite model. This allows for including 3D objects associated withhazardous site conditions among the features for each site condition.For extraction of text labels, the pre-processing may include convertingvoice data to text, cleaning text data, or both. The cleaning mayinclude, for example, modifying identified code numbers to match apredetermined format, correcting typos (e.g., replacing “$” in a codenumber with “4”), or both.

At S330, the extracted features and labels are input to a machinelearning algorithm to train a classifier. The trained classifier may beconfigured to classify site conditions into, for example, hazardous ornon-hazardous, a particular type of hazardous site condition, aseverity, a likelihood of accident, or a combination thereof. In someembodiments, multiple classifiers may be trained. For example, a firstclassifier may be trained to classify whether a site condition ishazardous or not, and a second classifier may be trained to classify thesite condition into a specific type of hazardous site condition. In afurther embodiment, site conditions may be analyzed by the firstclassifier, and only site conditions classified as hazardous areanalyzed by the second classifier. This increases efficiency ofclassification by reducing number of sets of features needed to beclassified with respect to each potential classification (e.g., whetherhazardous or not, type of hazard, severity, and likelihood) when, forexample, only a small proportion of site conditions are hazardous.

It should be noted that, in some implementations, additional featuresand labels may be input to the machine learning algorithm to update orotherwise retrain the classifier. This allows for updatingclassifications based on, for example, new visual identifiers ofviolations, new regulations, and the like.

At S340, when the classifier is trained, an enhanced floor plan forwhich hazardous site conditions are to be identified is received. Thereceived enhanced floor plan includes visual multimedia content, includeenvironmental data, or both. The received enhanced floor plan does notneed to include annotations. Rather, annotations may be provided by asite inspector based on a notification indicating hazardous siteconditions identified through using the classifier.

At S350, features are extracted from the enhanced floor plan. Thefeatures may be extracted as described herein above with respect toS320.

At S360, the extracted features are input to the classifier and theclassifier outputs one or more classifications.

At S370, based on the output classifications, one or more hazardous siteconditions are identified. In an embodiment, S370 may further includeproviding, but is not limited to, locations of the hazardous siteconditions, types of hazardous site conditions identified, regulatoryrequirements violated, severity, likelihood of accident, a combinationthereof, and the like. The identification is based on the point on theenhanced floor plan associated with each site condition classified ashazardous or a particular type of hazardous site condition.

At S380, a notification indicating the identified hazardous siteconditions is generated and sent to, for example, a user device of asite inspector. In some implementations, S380 may include addingannotations to the enhanced floor plan based on the identified hazardoussite conditions. In another implementation, the identified hazardoussite conditions may be shown on a dashboard (e.g., to a site inspector)and, and a notification may be sent only if the identified hazardoussite condition is confirmed as hazardous. The notification may includethe enhanced floor plan with added annotations or a link thereto. Insome implementations, the notification may also include electronicdocuments related to the regulations or other standards that areviolated by each identified hazardous site condition, or links thereto.

As a non-limiting example, an enhanced floor plan is generated based ondata collected on a jobsite. An image of the enhanced floor plan shows aworker that is not properly secured in a harness while suspended morethan 6 feet off of the ground. The site inspector or risk engineer flags(annotates) the worker shown in the image as a hazardous site conditionby drawing a boundary box around the worker and associates theannotation with a particular OSHA or other safety guideline violationvia standard semantics and text included as metadata for the image.Visual features are extracted from the image and labels are extractedfrom the annotations for the image.

The extracted features and labels are input, along with other visualfeatures and respective labels of visual and non-visual multimediacontent of the same and different enhanced floor plans, to a machinelearning algorithm to train a classifier. When a subsequently receivedenhanced floor plan to be analyzed for hazardous site conditions isreceived, visual features are extracted from images or videos from thesubsequently received enhanced floor plan and input to the classifier.If the subsequently received enhanced floor plan includes an image of aworker that is not secured in a harness while suspended more than 6feet, the classifier may classify the site condition shown in the imageas failure to secure worker in violation of a specific OSHA regulation.

FIG. 4 is an example schematic diagram of the hazard identifier 120according to an embodiment. The hazard identifier 120 includes aprocessing circuitry 410 coupled to a memory 415, a storage 420, anetwork interface 430, and a machine learning module 440. In anotherembodiment, the components of the hazard identifier 120 may becommunicatively connected via a bus 450.

The processing circuitry 410 may be realized as one or more hardwarelogic components and circuits. For example, and without limitation,illustrative types of hardware logic components that can be used includefield programmable gate arrays (FPGAs), application-specific integratedcircuits (ASICs), Application-specific standard products (ASSPs),system-on-a-chip systems (SOCs), general-purpose microprocessors,microcontrollers, digital signal processors (DSPs), and the like, or anyother hardware logic components that can perform calculations or othermanipulations of information.

The memory 415 may be volatile (e.g., RAM, etc.), non-volatile (e.g.,ROM, flash memory, etc.), or a combination thereof. In oneconfiguration, computer readable instructions to implement one or moreembodiments disclosed herein may be stored in the storage 420.

In another embodiment, the memory 415 is configured to store software.Software shall be construed broadly to mean any type of instructions,whether referred to as software, firmware, middleware, microcode,hardware description language, or otherwise. Instructions may includecode (e.g., in source code format, binary code format, executable codeformat, or any other suitable format of code). The instructions, whenexecuted by the one or more processors, cause the processing circuitry410 to perform the various processes described herein. Specifically, theinstructions, when executed, cause the processing circuitry 410 toperform the embodiments described herein.

The storage 420 may be magnetic storage, optical storage, and the like,and may be realized, for example, as flash memory or other memorytechnology, CD-ROM, Digital Versatile Disks (DVDs), or any other mediumwhich can be used to store the desired information.

The network interface 430 allows the hazard identifier 120 tocommunicate with the user device 130, the database 140, the data sources150, or a combination of, for the purpose of, for example, retrievingenhanced floor plans, retrieving regulatory requirements and othersafety standards, sending notifications of identified hazards, and thelike.

The machine learning module 440 is configured to train a classifier tobe utilized for classifying potential hazardous site conditions. Theinputs utilized to train the classifiers include visual features andannotation features, and may further include environmental features.

It should be understood that the embodiments described herein are notlimited to the specific architecture illustrated in FIG. 4, and otherarchitectures may be equally used without departing from the scope ofthe disclosed embodiments.

FIG. 5 is an example flowchart 500 illustrating a method for creating anenhanced floor plan according to an embodiment. In an embodiment, themethod is performed by a floor plan enhancer, for example, the floorplan enhancer described further in U.S. patent application Ser. No.16/220,709, the contents of which are hereby incorporated by reference.

At S510, visual multimedia content showing a site is received. Thevisual multimedia content may be 2D or 3D, and may include, but is notlimited to, images, video, or both. When the visual multimedia contentincludes 2D images, 3D characteristic data such as LIDAR measurementsmay also be received. In some implementations, S510 may further includereceiving an input floor plan or a floor identifier that may be utilizedto retrieve a floor plan associated with the floor identifier.

At S520, the received visual multimedia content is converted into asequence of frames. Each frame is a 2D image. When the visual multimediacontent includes 3D content, two or more frames may be generated foreach 3D image or each of at least one portion of a 3D video. The two ormore frames generated for 3D content may include 2D images thatcollectively show the entire view illustrated in the 3D content. Whenthe visual multimedia content includes 3D video, converting the visualmultimedia content into a sequence of frames may include identifyingportions of the 3D video, then converting each 3D video portion into twoor more 2D images.

At S530, based on the sequence of frames, a sparse 3D model of the siteis generated. The sparse 3D model is a 3D representation of the siteincluding a point cloud including reference points representing portionsof any objects or other visual features illustrated in the visualmultimedia content.

In an example implementation, the visual multimedia content includes 2Dimages, the sparse 3D model may be generated based on a sequence offrames including the 2D images and 3D characteristic data captured byone or more sensors.

At S540, the sparse 3D model is geo-localized with respect to an input2D or 3D floor plan model. In an embodiment, S540 includes eitherconverting the sparse 3D model to a 2D site model and matching featuresof the 2D site model to the input 2D floor plan model, or matchingfeatures of the sparse 3D model to features of a 3D building informationmodel.

In an example implementation, the feature matching may be based onmatching of a distinct set of features that uniquely identify a portionof the site between the 2D site model and the 2D floor plan model. Eachfeature may be matched with respect to a number of reference points perfeature. The features may include, but are not limited to, corners,columns, walls, and the like, depending on various phases of theconstruction. Based on the matching features, an orientation of the 2Dsite model relative to the 2D floor plan model is determined. As anon-limiting example, a corner of a building may be a feature having 4reference points. Matching the corner as shown in the 2D model to thecorner as shown in the 2D floor plan model allows for determining theorientation of the 2D site model to that of the 2D floor plan model, forexample, a 90-degree clockwise rotation of the corner as seen in the 2Dsite model relative to the corner as shown in the 2D floor plan modelindicates that the 2D site model is oriented 90 degrees clockwise fromthe 2D floor plan model.

At S550, an enhanced floor plan is created. Creating the enhanced floorplan may include superimposing the matched sparse 3D model to the 3Dbuilding information model. Alternatively, creating the enhanced floorplan may include superimposing the mapped 2D site model to the 2D floorplan model. To this end, S550 may further include generating a graphicaluser interface (GUI) including the enhanced floor plan. One or morepoints on the enhanced floor plan may be associated with correspondingportions of the visual multimedia content that were, for example,captured at the respective points. Interacting with the GUI may allow,for example, adding comments and annotations, viewing portions of thevisual multimedia content, viewing historical data of the enhanced floorplan (“time travel”), and the like.

FIGS. 6A-6C are example screenshots 600A, 600B, and 600C, respectively,of graphical user interfaces illustrating an example enhanced floor planaccording to an embodiment. In the example screenshot 600A, awalkthrough path 610 is superimposed on a 2D floor plan model 620. Thewalkthrough path 610 includes a set of points that collectivelydemonstrate a route taken during a walkthrough of a floor shown in the2D floor plan model.

In the example screenshot 600B, an image icon 630 has been superimposedon the walkthrough path 610 and the 2D floor plan model 620. The imageicon 630 is associated with one of the points of the walkthrough path610, and may be superimposed in response to selection of the associatedpoint (e.g., by a user clicking on the point via a graphical userinterface including the walkthrough path 610 superimposed on the 2Dfloor plan model 620).

In the example screenshot 600C, an image 640 of a site is displayed, forexample in response to an interaction with one of the points on thewalkthrough path 610. The image 640 includes annotations such asboundary boxes 650-1 through 650-3 and corresponding textual annotations660-1 through 660-3. The boundary boxes 650-1 through 650-3 surroundobjects in the image 640 that may be potential hazards, and eachrespective textual annotation 660-1 through 660-3 provides a textualdescription of, for example, whether the object demonstrates a hazard, atype of hazard, a severity of the hazard, a likelihood of injury for thehazard, a combination thereof, and the like. In some implementations(not shown), the boundary boxes 650, the textual annotations 660, orboth, may be depicted in different colors based on, for example, theseverity or likelihood of injury of a hazard.

The graphical user interface may be interacted with in order to selectportions of the enhanced floor plan and to input annotations for theselected portions. In some implementations, the image icon 630 may alsobe interacted with in order to, for example, cause display of acorresponding image, view annotations associated with the correspondingimage, view historical images showing the same or similar point on thewalkthrough path 610, and the like. The same or similar point may be,for example, a point captured from the same position or approximatelythe same position (e.g., within a predetermined threshold distance).

It should be noted that the method for creating enhanced floor plans isdescribed with respect to creating an enhanced floor plan for a sitebased on visual content captured during a walkthrough merely forsimplicity purposes and without limitation on the disclosed embodiments.In an example implementation, the site is a construction site. The sitemay be any location or portion thereof such as, but not limited to, afloor of a building under construction, a room of a building underconstruction, an outdoor location in which structures are erected (e.g.,a location of a street fair in which tents and amusement rides areconstructed, a location of a park including statues and playgroundequipment, etc.), a highway or other portion of road, virtual versionsthereof (e.g., a floor of a building to be rendered via virtual realityprograms), and the like.

It should be also noted that the method for creating enhanced floorplans is described with respect to 2D floor plan models merely forsimplicity purposes and without limitation on the disclosed embodiments.The disclosed embodiments may be equally applicable to other locations,regardless of whether they are typically characterized as a “floor.” Asa non-limiting example, the disclosed embodiments may be utilized toprovide an enhanced map of an outdoor site rather than a floor of abuilding.

The various embodiments disclosed herein can be implemented as hardware,firmware, software, or any combination thereof. Moreover, the softwareis preferably implemented as an application program tangibly embodied ona program storage unit or computer readable medium consisting of parts,or of certain devices and/or a combination of devices. The applicationprogram may be uploaded to, and executed by, a machine comprising anysuitable architecture. Preferably, the machine is implemented on acomputer platform having hardware such as one or more central processingunits (“CPUs”), a memory, and input/output interfaces. The computerplatform may also include an operating system and microinstruction code.The various processes and functions described herein may be either partof the microinstruction code or part of the application program, or anycombination thereof, which may be executed by a CPU, whether or not sucha computer or processor is explicitly shown. In addition, various otherperipheral units may be connected to the computer platform such as anadditional data storage unit and a printing unit. Furthermore, anon-transitory computer readable medium is any computer readable mediumexcept for a transitory propagating signal.

It should be understood that any reference to an element herein using adesignation such as “first,” “second,” and so forth does not generallylimit the quantity or order of those elements. Rather, thesedesignations are generally used herein as a convenient method ofdistinguishing between two or more elements or instances of an element.Thus, a reference to first and second elements does not mean that onlytwo elements may be employed there or that the first element mustprecede the second element in some manner. Also, unless statedotherwise, a set of elements comprises one or more elements.

As used herein, the phrase “at least one of” followed by a listing ofitems means that any of the listed items can be utilized individually,or any combination of two or more of the listed items can be utilized.For example, if a system is described as including “at least one of A,B, and C,” the system can include A alone; B alone; C alone; A and B incombination; B and C in combination; A and C in combination; or A, B,and C in combination.

All examples and conditional language recited herein are intended forpedagogical purposes to aid the reader in understanding the principlesof the disclosed embodiment and the concepts contributed by the inventorto furthering the art, and are to be construed as being withoutlimitation to such specifically recited examples and conditions.Moreover, all statements herein reciting principles, aspects, andembodiments of the disclosed embodiments, as well as specific examplesthereof, are intended to encompass both structural and functionalequivalents thereof. Additionally, it is intended that such equivalentsinclude both currently known equivalents as well as equivalentsdeveloped in the future, i.e., any elements developed that perform thesame function, regardless of structure.

What is claimed is:
 1. A method for identifying hazardous siteconditions, comprising: training a classifier using a labeled trainingset, wherein the classifier is trained to classify site conditions basedon features extracted from enhanced floor plans, wherein the labeledtraining set includes a plurality of training features and a pluralityof training labels, wherein the plurality of training features areextracted from a plurality of training enhanced floor plans, wherein theplurality of training labels are a plurality of hazardous site conditionidentification labels; and identifying at least one hazardous sitecondition of a test enhanced floor plan by applying the classifier to aplurality of test features extracted from the test enhanced floor plan,wherein the classifier outputs at least one site conditionclassification when applied to the plurality of test features, whereinthe at least one hazardous site condition is identified based on the atleast one site condition classification.
 2. The method of claim 1,wherein each of the plurality of training enhanced floor plansrepresents a training site, wherein each of the plurality of trainingenhanced floor plans includes at least one temporal variation, whereineach temporal variation of each training enhanced floor plandemonstrates a state of the training site represented by the trainingenhanced floor plan at a different point in time.
 3. The method of claim1, wherein the classifier is a first classifier, further comprising:applying a second classifier to a subset of the plurality of testfeatures based on the identified at least one hazardous site condition,wherein the second classifier is trained to classify types of hazardoussite conditions, wherein the subset of the plurality of test features isrelated to the identified at least one hazardous site condition.
 4. Themethod of claim 1, wherein the plurality of training labels is extractedfrom a plurality of annotations of the plurality of training enhancedfloor plans, wherein each of the plurality of annotations is associatedwith a training hazardous site condition of one of the plurality oftraining enhanced floor plans.
 5. The method of claim 1, wherein each ofthe at least one site condition classification indicates whether a sitecondition is hazardous.
 6. The method of claim 1, wherein each of the atleast one site condition classification indicates a type of hazard. 7.The method of claim 1, wherein each of the at least one hazardous sitecondition classification indicates a severity of hazard of a sitecondition.
 8. The method of claim 1, wherein the test enhanced floorplan is created based on a geo-localization of a sparsethree-dimensional (3D) model with respect to a site layout model,wherein the site layout model includes a plurality of floor plan points,wherein the geo-localization of the sparse 3D model includes identifyinga plurality of matching site layout model features of the site layoutmodel with respect to the sparse 3D model.
 9. The method of claim 8,wherein each of the site layout model features is a distinct feature setidentified in the site layout model, wherein each distinct feature setdefines a site layout model feature with respect to an arrangement ofreference points, wherein each arrangement of reference points for asite layout model feature does not change based on an orientation of thesite layout model feature.
 10. A non-transitory computer readable mediumhaving stored thereon instructions for causing a processing circuitry toexecute a process, the process comprising: training a classifier using alabeled training set, wherein the classifier is trained to classify siteconditions based on features extracted from enhanced floor plans,wherein the labeled training set includes a plurality of trainingfeatures and a plurality of training labels, wherein the plurality oftraining features are extracted from a plurality of training enhancedfloor plans, wherein the plurality of training labels are a plurality ofhazardous site condition identification labels; and identifying at leastone hazardous site condition of a test enhanced floor plan by applyingthe classifier to a plurality of test features extracted from the testenhanced floor plan, wherein the classifier outputs at least one sitecondition classification when applied to the plurality of test features,wherein the at least one hazardous site condition is identified based onthe at least one site condition classification.
 11. A system foridentifying hazardous site conditions, comprising: a processingcircuitry; and a memory, the memory containing instructions that, whenexecuted by the processing circuitry, configure the system to: train aclassifier using a labeled training set, wherein the classifier istrained to classify site conditions based on features extracted fromenhanced floor plans, wherein the labeled training set includes aplurality of training features and a plurality of training labels,wherein the plurality of training features are extracted from aplurality of training enhanced floor plans, wherein the plurality oftraining labels are a plurality of hazardous site conditionidentification labels; and identify at least one hazardous sitecondition of a test enhanced floor plan by applying the classifier to aplurality of test features extracted from the test enhanced floor plan,wherein the classifier outputs at least one site conditionclassification when applied to the plurality of test features, whereinthe at least one hazardous site condition is identified based on the atleast one site condition classification.
 12. The system of claim 11,wherein each of the plurality of training enhanced floor plansrepresents a training site, wherein each of the plurality of trainingenhanced floor plans includes at least one temporal variation, whereineach temporal variation of each training enhanced floor plandemonstrates a state of the training site represented by the trainingenhanced floor plan at a different point in time.
 13. The system ofclaim 11, wherein the classifier is a first classifier, wherein thesystem is further configured to: apply a second classifier to a subsetof the plurality of test features based on the identified at least onehazardous site condition, wherein the second classifier is trained toclassify types of hazardous site conditions, wherein the subset of theplurality of test features is related to the identified at least onehazardous site condition.
 14. The system of claim 11, wherein theplurality of training labels is extracted from a plurality ofannotations of the plurality of training enhanced floor plans, whereineach of the plurality of annotations is associated with a traininghazardous site condition of one of the plurality of training enhancedfloor plans.
 15. The system of claim 11, wherein each of the at leastone site condition classification indicates whether a site condition ishazardous.
 16. The system of claim 11, wherein each of the at least onesite condition classification indicates a type of hazard.
 17. The systemof claim 11, wherein each of the at least one hazardous site conditionclassification indicates a severity of hazard of a site condition. 18.The system of claim 11, wherein the test enhanced floor plan is createdbased on a geo-localization of a sparse three-dimensional (3D) modelwith respect to a site layout model, wherein the site layout modelincludes a plurality of floor plan points, wherein the geo-localizationof the sparse 3D model includes identifying a plurality of matching sitelayout model features of the site layout model with respect to thesparse 3D model.
 19. The system of claim 18, wherein each of the sitelayout model features is a distinct feature set identified in the sitelayout model, wherein each distinct feature set defines a site layoutmodel feature with respect to an arrangement of reference points,wherein each arrangement of reference points for a site layout modelfeature does not change based on an orientation of the site layout modelfeature.
 20. A method for identifying hazardous site conditions,comprising: identifying at least one hazardous site condition of a testenhanced floor plan by applying a classifier to a plurality of testfeatures extracted from the test enhanced floor plan, wherein theclassifier is trained to classify site conditions using a labeledtraining set, wherein a labeled training set includes a plurality oftraining features and a plurality of training labels, wherein theplurality of training features are extracted from a plurality oftraining enhanced floor plans, wherein the plurality of training labelsare a plurality of hazardous site condition identification labels,wherein the classifier outputs at least one site conditionclassification when applied to the plurality of test features, whereinthe at least one hazardous site condition is identified based on the atleast one site condition classification.
 21. The method of claim 1,wherein the identified at least one hazardous site condition indicatesat least one hazard at a construction site.
 22. The method of claim 1,wherein the identified at least one hazardous site condition includes atleast one of: a missing guardrail, an unsecured person suspended abovethe ground, a slippery substance on the ground, and a hazardoussubstance located within a predetermined distance of another object.